Papers with attention uncertainty
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Enabling LLMs to handle lengthy context is currently a research hotspot . a notable challenge limiting further customization is the inability of LLM to utilize context beyond pretrained length due to the inherent flaw of rotary position embedding (RoPE). |
| Approach: | They propose to extend the RoPE from an attention perspective and on two benchmarking tasks. |
| Outcome: | The proposed extension of the RoPE improves extrapolation and retrieval errors. |